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AUTOMATED DETECTION OF OBJECTS IN A BIOLOGICAL SAMPLE 

CLAIM OF PRIORITY 
5 This application claims the benefit of priority, 

under 35 USC §119(e) (1), of U.S. provisional 
application 60/143,823, filed July 13, 1999, and this 
application is a continuation of U.S. Patent 
Application Serial No. 08/758,436, filed November 27, 
10 1996, which claims the benefit of U.S. Provisional 
Application No. 60/026,805, filed on November 30, 
1995. 

TECHNICAL FIELD 
15 The invention relates generally to light 

microscopy and, more particularly, to automated light 
microscopic methods and an apparatus for detection of 
proteins associated with cell proliferative 
disorders. 
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BACKGROUND OF THE INVENTION 
In the field of medical diagnostics including 
oncology, the detection, identification, quantitation 
and characterization of cells of interest, such as 
5 cancer cells, through testing of biological specimens 
is an important aspect of diagnosis. Typically, a 
biological specimen such as bone marrow, lymph nodes, 
peripheral blood, cerebrospinal fluid, urine, 
effusions, fine needle aspirates, peripheral blood 

10 scrapings or other materials are prepared by staining 
the specimen to identify cells of interest. One 
method of cell specimen preparation is to react a 
specimen with a specific probe which can be a 
monoclonal antibody, a polyclonal antiserum, or a 

15 nucleic acid which is reactive with a component of 
the cells of interest, such as tumor cells. The 
reaction may be detected using an enzymatic reaction, 
such as alkaline phosphatase or glucose oxidase or 
peroxidase to convert a soluble colorless substrate 

20 to a colored insoluble precipitate, or by directly 
conjugating a dye to the probe. Examination of 
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biological specimens in the past has been performed 
manually by either a lab technician or a pathologist. 
In the manual method, a slide prepared with a 
biological specimen is viewed at a low magnification 
5 under a microscope to visually locate candidate cells 
of interest. Those areas of the slide where cells of 
interest are located are then viewed at a higher 
magnification to confirm those objects as cells of 
interest, such as tumor or cancer cells. The manual 

10 method is time consuming and prone to error including 
missing areas of the slide. Automated cell analysis 
systems have been developed to improve the speed and 
accuracy of the testing process. One known 
interactive system includes a single high power 

15 microscope objective for scanning a rack of slides, 
portions of which have been previously identified for 
assay by an operator. In that system, the operator 
first scans each slide at a low magnification similar 
to the manual method and notes the points of interest 

20 on the slide for later analysis. The operator then 
stores the address of the noted location and the 
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associated function in a data file. Once the points 
of interest have been located and stored by the 
operator, the slide is then positioned in an 
automated analysis apparatus which acquires images of 
5 the slide at the marked points and performs an image 
analysis. 

A number of cellular proteins are related to 
cell proliferation and cell signaling. Many of these 
proteins are critical for normal cell growth. For 

10 example, HER2 (neu) is a growth factor receptor and 
when found within tumor cells amounts to an 
aggressively growing tumor. Studies have determined 
that a significantly decreased disease- free survival 
and overall survival of a patient with over- 

15 expression of HER2. Before an oncologist prescribes 
an anti-HER2/neu therapeutic, an immunohistochemistry 
(IHC) assessment for HER2 / neu is desirable. 
Therapeutic availability increases the need for a 
standard methodology for assessing the expression of 

20 HER2/neu. 
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Three general methods are currently available 
for the detection of HER2/neu: genetic detection, 
protein expression, and protein activity. In situ 
hybridization methods are typically used for HER2/neu 
5 genetic detection- Immunohistochemistry methods are 
used for the assessment of HER2/neu protein 
expression. 

SUMMARY OF THE INVENTION 
10 A problem with the foregoing automated system is 

the continued need for operator input to initially 
locate cell objects for analysis. Such continued 
dependence on manual input can lead to errors 
including cells of interest being missed. Such 
15 errors can be critical especially in assays for so- 
called rare events, e.g., finding one tumor cell in a 
cell population of one million normal cells. 
Additionally, manual methods can be extremely time 
consuming and can require a high decree of training 
20 to identify and/or quantify cells. This is not only 
true for tumor cell detection, but also for other 
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applications ranging from neutrophil alkaline 
phosphatase assays, reticulocyte counting and 
maturation assessment, and others. The associated 
manual labor leads to a high cost for these 
5 procedures in addition to the potential errors that 
can arise from long, tedious manual examinations. A 
need exists, therefore, for an improved automated 
cell analysis system which can quickly and accurately 
scan large amounts of biological material on a slide. 

10 Accordingly, the present invention provides a method 
and apparatus for automated cell analysis which 
eliminates the need for operator input to locate cell 
objects for analysis. 

In accordance with the present invention, a 

15 slide prepared with a biological specimen and reagent 
is placed in a slide carrier which preferably holds 
four slides. The slide carriers are loaded into an 
input hopper of the automated system. The operator 
may then enter data identifying the size, shape, and 

20 location of a scan area on each slide, or, 

preferably, the system automatically locates a scan 
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area for each slide during slide processing. The 
operator then activates the system for slide 
processing. At system activation, a slide carrier is 
positioned on an X-Y stage of an optical system. Any 
5 bar codes used to identify slides are then read and 
stored for each slide in a carrier. The entire slide 
is rapidly scanned at a low magnification, typically 
lOx. At each location of the scan, a low 
magnification image is acquired and processed to 

10 detect candidate objects of interest. Preferably, 
color, size, and shape are used to identify objects 
of interest. The location of each candidate object 
of interest is stored. 

At the completion of the low level scan for each 

15 slide in the carrier on the stage, the optical system 
is adjusted to a high magnification such as 40x or 
60x, and the X-Y stage is positioned to the stored 
locations for the candidate objects of interest on 
each slide in the carrier. A high magnification 

20 image is acquired for each candidate object of 

interest and a series of image processing steps are 
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performed to confirm the analysis which was performed 
at low magnification. A high magnification image is 
stored for each confirmed object of interest. 

Additionally, control slides including positive 
5 and negative controls, may be used to determine 

background staining. For example, positive control 
slides for a particular staining technique can be run 
followed by a negative control in order to determine 
a delta for the controls. Subsequent scanning for 
10 objects of interest may then differentiate such 

objects based upon their color or intensity above the 



These images are then available for retrieval by 
a pathologist or cytotechnologist to review for final 

15 diagnostic evaluation. Having stored the location of 
each object of interest, a mosaic comprised of the 
candidate objects of interest for a slide may be 
generated and stored. The pathologist or 
cytotechnologist may view the mosaic or may also 

20 directly view the slide at the location of an object 
of interest in the mosaic for further evaluation. 



delta. 
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The mosaic may be stored on magnetic media for future 
reference or may be transmitted to a remote site for 
review and/or storage. The entire process involved 
in examining a single slide takes on the order of 2- 
5 15 minutes depending on scan area size and the number 
of detected candidate objects of interest. 

The present invention has utility in the field 
of oncology for the early detection of minimal 
residual disease ( % 'micrometastases" ) . Other useful 

10 applications include prenatal diagnosis of fetal 

cells in maternal blood and in the field of i 
infectious diseases to identify pathogens and viral 
loads, alkaline phosphatase assessments, reticulocyte 
counting, and others. The processing of images 

15 acquired in the automated scanning of the present 
invention preferably includes the steps of 
transforming the image to a different color space; 
filtering the transformed image with a low pass 
filter; dynamically thresholding the pixels of the 

20 filtered image to suppress background material; 
performing a morphological function to remove 
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artifacts from the thresholded image; analyzing the 
thresholded image to determine the presence of one or 
more regions of connected pixels having the same 
color; and categorizing every region having a size 
5 greater than a minimum size as a candidate object of 
interest . 

According to another aspect of the invention, 
the scan area is automatically determined by scanning 
the slide; acquiring an image at each slide position; 

10 analyzing texture information of each image to detect 
the edges of the specimen; and storing the locations 
corresponding to the detected edges to define the 
scan area. According to yet another aspect of the 
invention, automated focusing of the optical system 

15 is achieved by initially determining a focal plane 
from an array of points or locations in the scan 
area. The derived focal plane enables subsequent 
rapid automatic focusing in the low power scanning 
operation. The focal plane is determined by 

20 determining proper focal positions across an array of 
locations and performing an analysis such as a least 
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squares fit of the array of focal positions to yield 
a focal plane across the array. Preferably, a focal 
position at each location is determined by 
incrementing the position of a Z stage for a fixed 
5 number of coarse and fine iterations. At each 

iteration, an image is acquired and a pixel variance 
or other optical parameter about a pixel mean for the 
acquired image is calculated to form a set of 
variance data. A least squares fit is performed on 

10 the variance data according to a known function. The 
peak value of the least squares fit curve is selected 
as an estimate of the best focal position. 

In another aspect of the present invention, 
another focal position method for high magnification 

15 locates a region of interest centered about a 

candidate object of interest within a slide which 
were located during an analysis of the low 
magnification images. The region of interest is 
preferably n columns wide, where n is a power of 2. 

20 The pixels of this region are then processed using a 
Fast Fourier Transform to generate a spectra of 
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component frequencies and corresponding complex 
magnitude for each frequency component. Magnitudes 
of the frequency components which range from 25% to 
75% of the maximum frequency component are squared 
5 and summed to obtain the total power for the region 
of interest . This process is repeated for other Z 
positions and the Z position corresponding to the 
maximum total power for the region of interest is 
selected as the best focal position. 

10 According to still another aspect of the 

invention, a method and apparatus for automated slide 
handling is provided. A slide is mounted onto a 
slide carrier with a number of other slides side -by- 
side. The slide carrier is positioned in an input 

15 feeder with other slide carriers to facilitate 

automatic analysis of a batch of slides. The slide 
carrier is loaded onto the X-Y stage of the optical 
system for the analysis of the slides thereon. 
Subsequently, the first slide carrier is unloaded 

20 into an output feeder after automatic image analysis 
and the next carrier is 
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automatically loaded. 

In a specific embodiment the invention provides 
an automated system for the quantitation of proteins 
associated with cell proliferative disorders, such as 
5 HER2/neu expression in tissue. The invention is 
useful to determine the over-expression of HER2 in 
tissue, especially breast tissue. 



DESCRIPTION OF THE DRAWINGS 
10 The above and other features of the invention 

including various novel details of construction and 
combinations of parts .will now be more particularly 
described with reference to the accompanying drawings 
and pointed out in the claims . It will be understood 
15 that the particular apparatus embodying the invention 
is shown by way of illustration only and not as a 
limitation of the invention. The principles and 
features of this invention may be employed in varied 
and numerous embodiments without departing from the 
20 scope of the invention. 
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Fig. 1 is a perspective view of an apparatus for 
automated cell analysis embodying the present 
invention. 

Fig. 2 is a block diagram of the apparatus shown 
5 in Fig . 1 . 

Fig. 3 is a block diagram of the microscope 
controller of Fig. 2. 

Fig. 4 is a plan view of the apparatus of Fig. 1 
having the housing removed. 
10 Fig. 5 is a side view of a microscope subsystem 

of the apparatus of Fig . 1 . 

Fig; 6a is a top view of a slide carrier for use 
with the apparatus of Fig. 1. 

Fig. 6b is a bottom view of the slide carrier of 
15 Fig. 6a. 

Fig. 7a is a top view of an automated slide 
handling subsystem of the apparatus of Fig. 1. 

Fig. 7b is a partial cross-sectional view of the 
automated slide handling subsystem of Fig. 7a taken 
20 on line A- A. 



WO 01/04828 




PCT/USOO/19046 



15 



Fig. 8 is an end view of the input module of the 
automated slide handling subsystem. 8a- 8d illustrate 
the input operation of the automatic slide handling 
subsystem. 

5 Fig. 9a- 9d illustrate the output operation of 

the automated slide handling subsystem. 

Fig. 10 is a flow diagram of the procedure for 
automatically determining a scan area. 

Fig. 11 shows the scan path on a prepared slide 
10 in the procedure of Fig. 10. 

Fig. 12 illustrates an image of a field acquired - 
in the procedure of Fig. 10. 

Fig. 13A is a flow diagram of a preferred 
procedure of redetermining focal position. 
15 Fig. 13B is a flow diagram of a preferred 

procedure for determining a focal position for 
neutrophils stained with Fast Red and count erstained 
with hemotoxylin. 

Fig. 14 is a flow diagram of a procedure for 
20 automatically determining initial focus. 
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Fig. 15 shows an array of slide positions for 
use in the procedure of Fig. 14. 

Fig. 16 is a flow diagram of a procedure for 
automatic focusing at a high magnification. 
5 Fig. 17A is a flow diagram of an overview of the 

preferred process to locate and identify objects of 
interest in a stained biological specimen on a slide. 

Fig. 17B is a flow diagram of a procedure for 
color space conversion. 
10 Fig. 18 is a flow diagram of a procedure for 

background suppression via dynamic thresholding. 

Fig. 19 is a flow diagram of a procedure for 
morphological processing. 

Fig. 20 is a flow diagram of a procedure for 
15 blob analysis. 

Fig. 21 is a flow diagram of a procedure for 
image processing at a high magnification. 

Fig. 22 illustrates a mosaic of cell images 
produced by the apparatus. 
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Fig. 23 is a flow diagram of a procedure for 
estimating the number of nucleated cells in an 
objective field. 

Fig. 24 illustrates the apparatus functions 
5 available in a user interface of the apparatus. 



The invention provides an automated tissue image 



10 analysis method for the detection of a cell 

proliferative disorder, for example those associated 
with HER2/neu. Other automated tissue image analysis 
methods, such as immunohistochemical analyses of 
estrogen receptor and progesterone receptor are also 

15 provided. Other methods are within the scope of the 
invention, such as analyses for MM/MRD for tissue, 
MIB-I, Microvessel density analysis, the oncogene 
p53, immunophenotyping, hematoxylin/ eosin (H/E) 
morphological analysis, undifferentiated tumor 

20 classification, antibody titering, prominence of 
nucleoli, HIV p24, human papiloma virus (HPV; for 



DETAILED DESCRIPTION 



Overview 
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cervical biopsy) , and mitotic index; generally any 
diagnostic method utilizing staining techniques is 
within the scope of the present invention. The 
invention has utility in the field of oncology for 
5 the early detection of minimal residual disease 
( * 'micrometastases" ) . 

For the HER2/neu application, the Automated 
Cellular Imaging System (ACIS™) analyzes at least two 
subsamples: the H/E prepared slide and the 

10 immunohistochemistry prepared slide. A histological 
reconstruction of subsamples is displayed and used 
for the appropriate reviewing. An instrument 
automatically selects an area of interest, using 
color space transformations (and morphological 

15 filtering, if necessary) to identify and quantify the 
expression of the HER2/neu in the area of tissue 
selected from the immunohistochemistry prepared 
slide. The quantitative result, as well as any 
selected areas of interest from either slide, is 

20 incorporated into the patient report . 
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The following acronyms and terminology are used 
throughout this document: A worklist is a group of 
specimens that is prepared with the same staining 
techniques and typically analyzed with the same 
5 automated application. A specimen group consists of 
one or more patient cases. A histological 
reconstruction is an image of the whole specimen that 
has been mounted on a slide. This image is created 
by piecing together more than one fields of view at 
10 any objective. Unless otherwise defined, all 

technical and scientific terms used herein have the 
same meaning as commonly understood by one of 
ordinary skill in the art to which this invention 
belongs . 

15 Although methods and materials similar or 

equivalent to those described herein can be used in 
the practice or testing of the present invention, 
suitable methods and materials are described below. 
All publications, patent applications, patents, 

20 and other references mentioned herein are 

incorporated by reference in their entirety. In case 
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of conflict, the present application, including • 
definitions, will control. In addition, the 
materials and methods described herein are 
illustrative only and not intended to be limiting. 
5 Other features and advantages of the invention will 
be apparent from the following detailed description, 
the drawings, and from the claims. 



Preferred Workflow 
10 A preferred workflow is provided in the 

following outline: 

1. Specimens arrive in the laboratory. The 
specimens are labeled in accordance with 
standard hospital procedures to insure the 

15 chain of custody. 

2. Sample Preparation. 

2.1. Two or three tissue sections per 

patient case are preferable for this 
test . 

20 2.2. Mount one piece of the tissue to a 

slide, to be stained with H/E. The H/E 
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staining protocol is compatible with 
HER2/neu immunohistochemistry staining. 
Mount the second piece of the tissue to 
a slide, to be stained with the anti- 
HER2 immunohistochemistry staining 
system. 

If deemed necessary, mount a third 
piece of the tissue to a third slide, 
to be stained with a third antibody 
(i.e., patient negative control) and 
the anti-HER2 staining system. 
To avoid ACIS™ mechanical limitations, 
the specimen placement specifications 
is followed. Considering the frosted 
end of the slide as the top, specimen 
must be below 30 mm and above 65 mm 
from the top edge. The specimen may 
reach to the far side edges of the 
slide. 

Other patient samples? If yes, repeat 
section 2 . 
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2.7. Include two stain control slides. 
Sample Staining. 

3.1. Stain all H/E prepared slides with the 
H/E staining protocol. 

3.2. Stain all immunohistochemistry slides 
with anti-HER2 staining system. 

3.3. If deemed necessary, stain all negative 
immunohistochemistry slides with a 
third antibody (i.e., patient negative 
control) and the anti-HER2 staining 
system. 

3.4. Stain the positive and negative control 
slides with the anti-HER2 staining 
system and the appropriate antibodies . 

Barcode all prepared slides and place 
within open slots in the slide carriers. 
Manually study the positive and negative 
control slides and verify the staining is 
in control. Manually study the patient's 
positive and negative immunohistochemistry 
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slides and verify patient doesn't have an 
abnormality in background staining. 
Work- list entry 

6.1. Enter the information that is 
consistent per work-list. 

6.1.1. Specify the work- list 
identification. 

6.1.2. Specify the 
protocol/application (HER2/neu) 

6.1.3. Specify the barcodes of the 
positive and negative control 
slides 

6.2. Enter. the information that is related 
to each patient case. 

6.2.1. Accession number and Patient 
Identification ( ID) 

6.2.2. Specify the two/ three barcodes 
of the samples for this case (H/E, 
immunohistochemistry, patient 
negative control) 
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6.2.3. Patient specific information: 
Age (integer), Sex (M/F),Race 
(character - predefined 
selections) , Diagnosis (character 
- 2S6), Date of diagnosis (Date 
Format) , Date of last treatment 
(Date Format) , and Description of 
last treatment (character - 512) . 
If daily calibration of the instrument 
hasn't been completed, please do so before 
starting the work- list (5 min per 
calibration, necessary once per a day) . 
This calibration consists of radiometric, 
geometric, and color standard calibration. 
Load the carriers containing the control 
slides and patient cases into the input 
hopper . 

Start the automated analysis of the work- 
list. 

9.1. Should the stain controls not have a 
large enough delta A, the instrument 
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skips the rest of the slides within the 



worklist . 



10 



15 



10. After completion of the analysis, review 
the results of the work- list and construct 
the appropriate reports online. For each 
case, the low power H/E image appears 
automatically with the selection of the 
patient ID, accession number, or sample 
identification . 

10.1. While viewing the low power H/E 
Histological Reconstruction (HR) the 
user has the following choices: 

10.1.1. Select an area of the H/E HR 
to zoom on. 

10.1.2. Choose to go to the low power 
immunohistochemistry HR. 

10.1.3. Select an area to save within 
the pre-constructed report format. 

10.2. While viewing the high power H/E 
HR the user has the following choices: 
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10.2.1. Choose to go back to the low 
power H/E HR. 
.3. While viewing the low power 

immunohistochemistry HR the user has 
the following choices: 

10.3.1. Select an area of the 
immunohistochemistry HR to zoom 
on. 

10.3.2. Select an area of the low 
power immunohistochemistry HR and 
query for the -following 
quantitative results. The 
quantitative results are relative 
to the scoring of the positive 
stain control and the patient's 
negative immunohistochemistry 
slide. 

10.3.2.1. Over-expression within 
selected area reported as 
0, 1+, 2 + , or 3+. 
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10 . 3 . 2 . 2 . Percentage of positive 
cells within the selected 
area. 

.4. While viewing the high power 

immunohistochemistry HR the user has 

the following choices : 

10.4.1. Select an area of the high 

power immunohistochemistry HR and 
query for the following 
quantitative results. The 
quantitative results are relative 

r .to the scoring of the positive 
stain control and the patient's 
negative immunohistochemistry 
slide. 

10 • 4 . 1 . 1 . Over-expression within 
selected area reported as 
0, 1+, 2+, or 3+. 

10.4.1.2. Percentage of positive 
cells within the selected 
area. 
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10.4.2. Choose to go back to the low 
power immunohistochemistry HR. 

10.5. Other samples to review? 
10.5.1. If yes, repeat section 8. 

10.6. Should the analysis result in the 
tissue not being scanned, reload slide, 
open application, and adjust scan area 
to location of the tissue, if 
necessary. Should the scan area be 
adequate, but the focus points not fall 
on an area of tissue, resulting in 
failed focus, in the application change 
the focus inset to a number closer to 
one and confirm that the scan area is 
centered on an area of tissue. Execute 
the application. Repeat steps 10.1- 
10.5. 

10.7. Manually record results of 
analysis. 

Quality Assurance checks within the 
instrument . 
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11.1. At the completion of analyzing 
both the positive and negative control 
slides, the A between the scores are 
calculated. If this A isn't large 

5 enough, the analysis of all slides in 

this work- list is flagged accordingly 
and the instrument warns the user with 
an audible alarm. 

11.2. If the A between the patient's 
10 positive and negative 

immunohistochemistry slide isn't large 
enough, the patient is flagged 
accordingly . 

11.3. A montage of tissue sections from 
15 both the positive and negative 

controls. Five tissue section images 
from each are adequate. 
12. Preview the reports online and modify as 
necessary. 

20 13 . Send the reports to the printer or by the 

Internet to the requesting clinicians. The 
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output of the worklist is a stained 
positive control slide, a stained negative 
control slide, two stained slides per 
patient case (one H/E and another IHC) , and 
5 a comprehensive electronic report. The 

report includes the selected image 
sections, quantitative results, reference 
material, and other pertinent data. 



10 Sample Staining 

A cellular specimen (a "sample") is split to 
provide two or more subsamples. The term "sample" 
includes cellular material derived from a subject. 
Such samples include but are not limited to hair, 

15 skin samples, tissue sample, cultured cells, cultured 
cell media, and biological fluids. The term 
"tissue" refers to a mass of connected cells (e.g., 
CNS tissue, neural tissue, or eye tissue) derived 
from a human or other animal and includes the 

20 connecting material and the liquid material in 

association with the cells. The term "biological 
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fluid" refers to liquid material derived from a 
human or other animal- Such biological fluids 
include, but are not limited to, blood, plasma, 
serum, serum derivatives, bile, phlegm, saliva, 
5 sweat, amniotic fluid, and cerebrospinal fluid (CSF) , 
such as lumbar or ventricular CSF. The term 

sample" also includes media containing isolated 
cells. The quantity of sample required to obtain a 
reaction may be determined by one skilled in the art 
10 by standard laboratory techniques. The optimal 
quantity of sample may be determined by serial 
dilution. 

The HER2/neu method uses an anti-HER2/neu 
staining system, such as a commercially available 

15 kit, like that provided by DAKO (Carpinteria, CA) . 
The steps for the immunohistochemistry protocol are 
as follows: (1) Prepare wash buffer solution. 
(2) Deparaf f inize and rehydrate specimens. (3) 
Perform epitope retrieval. Incubate 40 min in a 95°C 

20 water bath. Cool slides for 20 min at room 

temperature. (4) Apply peroxidase blocking reagent. 
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Incubate 5 min. (5) Apply primary antibody or 
negative control reagent. Incubate 30 min +/- 1 min 
at room temperature. Rinse in wash solution. Place 
in wash solution bath. (6) Apply peroxidase labeled 
5 polymer. Incubate 30 min +/- 1 min at room 

temperature. Rinse in wash solution. Place in wash 
solution bath. (7) Prepare DAB substrate chromagen 
solution. (8) Apply substrate chromogen solution 
(DAB) . Incubate 5-10 min. Rinse with distilled 

10 water; (9) Counterstain. (10) Mount covers lips. The 
slide includes a cover- slip medium to protect the 
sample and to introduce optical correction consistent 
with microscope objective requirements. The 
coverslip covers the entire prepared specimen. 

15 Mounting the coverslip does not introduce air bubbles 
obscuring the stained specimen. This coverslip could 
potentially be a mounted 1-1/2 thickness coverslip 
with DAKO Ultramount medium. (11) A set of staining 
control slides are run with every worklist. The set 

20 includes a positive and negative control. The 
positive control is stained with the anti-HER2 
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antibody and the negative is stained with another 
antibody. Both slides are identified with a unique 
barcode. Upon reading the barcode, the instrument 
recognizes the slide as part of a control set, and 
5 runs the appropriate application. There may be one 
or two applications for the stain controls. (12) A 
set of instrument calibration slides includes the 
slides used for focus and color balance calibration. 
(13) A dedicated carrier is used for one- touch 

10 calibration. Upon successful completion of this 

calibration. '.procedure, the instrument reports itself 
to be calibrated: Upon successful completion of 
running the. standard slides, the user is able to 
determine whether the instrument is within standards 

15 and whether the inter- instrument and intra- instrument 
repeatability of test results. 

The hematoxylin/eosin (H/E) slides are prepared 
with a standard H/E protocol. Standard solutions 
include the following: (1) Gills hematoxylin 

20 (hematoxylin 6.0 g; aluminium sulphate 4.2 g; citric 
acid 1.4 g; sodium iodate 0.6 g; ethylene glycol 269 
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ml; distilled water 680 ml); (2) eosin (eosin 
yellowish 1.0 g; distilled water 100 ml); (3) lithium 
carbonate 1% (lithium carbonate 1 g; distilled water 
100 g) ; (4) acid alcohol 1% 70% (alcohol 99 ml cone; 
5 hydrochloric acid 1 ml); and (5) Scott's tap water. 
In a beaker containing 1 L distilled water, add 20 g 
sodium bicarbonate and 3.5 g magnesium sulphate. Add 
a magnetic stirrer and mix thoroughly to dissolve the 
salts. Using a filter funnel, pour the solution into 

10 a labeled bottle. 

The staining procedure is as follows: (1) Bring 
the sections to water; (2) place sections in ; t 

hematoxylin for 5 min; (3) wash in tap water; (4) 
'blue' the sections in lithium carbonate or Scott's 

15 tap water; (5) wash in tap water; (6) place sections 
in 1% acid alcohol for a few seconds; (7) wash in tap 
water; (8)place sections in eosin for 5 min; (9) wash 
in tap water; and (10) dehydrate, clear. Mount 
sections . 

20 The results of the H/E staining provide cells 

with nuclei stained blue-black, cytoplasm stained 
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varying shades of pink; muscle fibers stained deep 
pinky red; fibrin stained deep pink; and red blood 
cells stained orange-red. 

The invention also provides automated methods 
5 for analysis of estrogen receptor and progesterone 
receptor. The estrogen and progesterone receptors, 
like other steroid hormone receptors, plays a role in 
developmental processes and maintenance of hormone 
responsiveness in target cells. From the molecular 

10 viewpoint, estrogen and progesterone receptor 
interaction with target genes is of paramount 
importance in maintenance of normal cell function and 
is also involved in regulation, of mammary tumor cell 
function. The expression of progesterone receptor 

15 and estrogen receptor in breast tumors is a useful 
indicator for subsequent hormone therapy. An anti- 
estrogen receptor antibody labels epithelial cells of 
breast carcinomas which express estrogen receptor. 
An immunohistochemical assay of the estrogen receptor 

20 is performed using an ant i -estrogen receptor 

antibody, for example the well-characterized 1D5 
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clone, and the methods of Pertchuk, et al. (Cancer 
77: 2514-2519, 1996) or a commercially available 
immunohistochemistry system such as that provided by 
DAKO ( Carpenter ia CA; DAKO LSAB2 Immunostaining 
5 System) . 

In breast carcinoma cells, immunohistochemistry 
immunostaining of progesterone receptor has been 
demonstrated in the nuclei of cells from various 
histologic subtypes. An ant i -progesterone receptor 

10 antibody labels epithelial cells of breast carcinomas 
which express progesterone receptor. An 
immunohistochemical assay of the progesterone 
receptor is performed using an anti-estrogen receptor 
antibody, for example the well-characterized 1A6 

15 clone, and the methods of Pertchuk, et al. (Cancer 
77: 2514-2519, 1996) . 

Still other automated analyses are within the 
scope of the invention, including the following: 
Micrometastases/Metastatic Recurring Disease 

20 (MM /MRP) . Metastasis is the biological process 

whereby a cancer spreads to the distant part of the 
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body from its original site. A micrometastases is 
the presence of a small number of tumor cells, 
particularly in the lymph nodes and bone marrow. A 
metastatic recurring disease is similar to 
5 micrometastasis, but is detected after cancer therapy 
rather than before therapy. An immunohistochemical 
assay for MM/MRD is performed using a monoclonal 
antibody which reacts with an antigen (a metastatic- 
specific mucin) found in bladder, prostate and breast 
10 cancers . 

MIB-1 . MIB-1 is an antibody that can be used in 
immunohistochemical assays for the antigen Ki- 67. 
The clinical stage at first presentation is related 
to the proliferative index measured with Ki-67. High 

15 index values of Ki-67 are positively correlated with 
metastasis, death from neoplasia, low disease-free 
survival rates, and low overall survival rates. 

Microvessel Density Analysis . Microvessel 
density analysis is a measure of new blood vessel 

20 formation (angiogenesis) . Angiogenesis is 

characteristic of growing tumors. Intratumor 
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microvessel density can be assessed by anti-CD34 
immunostaining . 

The Oncogene p53 . Overexpression of the p53 
oncogene has been implicated as the most common 
5 genetic alteration in the development of human 
malignancies. Investigations of a variety of 
malignancies, including neoplasms of breast, colon, 
ovary, lung, liver, mesenchyme, bladder and myeloid, 
have suggested a contributing role of p53 mutation in 

10 the development of malignancy. The highest frequency 
of expression has been. demonstrated in tumors of the 
breast, colon and ovary. A .wide variety of normal 
cells do express a wildtype form of p53 but generally 
in restricted amounts. Overexpression and mutation 

15 of p53 have not been recognized in benign tumors or 
in normal tissue. An immunohistochemical assay of 
p53 is performed using an anti-p53 antibody, for 
example the well-characterized DO-7 clone. 
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Immunophenotyping . 

Undifferentiated tumor classification. 
Antibody titering. 

Prominence of Nucleoli . A nucleoli is an 
5 organelle in a cell nucleus. Cervical dysplasia 
refers to the replacement of the normal or 
metaplastic epithelium with atypical epithelial cells 
that have cytologic features that are pre-malignant 
(nuclear hyper chroma t ism, nuclear enlargement and 

10 irregular outlines, increased nuclear-to-cytoplasmic 
ratio, increased prominence of nucleoli) and 
chromosomal abnormalities. The changes seen in 
dysplastic cells are of the same kind but of a lesser 
degree than those of frankly malignant cells. In 

15 addition, there are degrees of dysplasia (mild, 
moderate, severe) . 

HIV p24 Protein . Human immunodeficiency virus 
(HIV) p24 antigen levels are measured in an 
immunohistochemistry assay using anti-HIV p24 

20 antigen. The HIV p24 protein test is used for the 
detection of HIV virus. The test can be used to 
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detect the virus, measure the amount of virus, and 

examine the virus' genetic composition. 

Human Papiloma Virus (HPV) . HPV is a sexually 

transmitted virus which causes warts. HPV is an 
5 important health concern because it may cause cancer 

of the cervix. An immunohistochemical assay of HPV 

is performed using an antibody associated with 

cervical cancer. Several serological responses are 

strongly associated with cervical cancer, notably the 
10 IgG response against the HPV 16 epitopes LI: 13, 

E2:9, and E7:5, and the IgA response against an HPV 

18 E2 -derived antigen. 

Mitotic Index . When a viral genome is : 

incorporated into the nuclear genetic material, 
15 uncontrolled malignant growth of the host cell may be 

promoted . 

Operating the Instrument 

While operating the instrument, the user 
20 interacts with a few screens including, but not 
limited to, Patient Entry, Review Data, Construct 
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Report, Manual Control, and User Preferences. Rather 
than the information entry revolving around the 
patient cases, the patient data entry interface is 
organized as worklist information composed of 
5 multiple patient cases. For example, a worklist for 
HER2 / neu is composed of tissue samples that are 
prepared in one of two methods, the HER2/neu 
immunohistochemistry staining technique or the H/E 
technique. The slides within a HER2 / neu worklist 

10 consists of a positive control slide (prepared with 
the immunohistochemistry technique and the HER2 
antibody) , a negative control slide (prepared with 
the immunohistochemistry. technique and another 
antibody) , and two slides for the one or more patient 

15 cases. Per patient case there are two slides 

prepared. One slide is prepared with the HER2 / neu 
immunohistochemistry technique and a second is 
prepared with the H/E technique for general tissue 
analysis . 

20 The data that is relevant for a HER2/neu 

worklist is the: (1) worklist name or identifier of 
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protocol (e.g., HER2/neu) ; barcode of the positive 
control slide; and barcode of the negative control 
slide. The data that is relevant on a per Patient 
Case is the: (4) patient ID; (5) accession number; 
5 (6) social security number; (7) referring hospital, 
(8) referring physician; (9) barcode of the 
immunohistochemistry prepared slide; and (10) barcode 
of the H/E prepared slide. 

The user starts analysis with the batch button 

10 and the slide carriers loaded into the input hopper. 
: The ; input hopper is composed of all the 

immunohistochemistry slides from the patient cases, 
all. the H/E slides from the patient cases, the 
positive control slide, and the negative control 

15 slide. During analysis the instrument reads the 
barcode on the slide and determines one of three 
situations and starts one of the three appropriate 
computer applications: (1) Slide is the positive 
control slide and run the HER2-positive application. 

20 (2) Slide is the negative control slide and run the 
HER2 -negative application. (3) Slide is either the 
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immunohistochemistry or H/E patient slide and run the 
HER2 application. 

The system reads industry standard barcodes. 
The barcode relates the type of protocol the 
5 instrument uses to analyze the slide. The type of 
data stored for each slide is determined by the stain 
preparation and the protocol used to analyze it. 
During the transport of the slide through the system, 
therefore, the chain of custody is preserved. When a 

10 barcode is unreadable or isn't found in one of the 

defined worklists, the action taken by the instrument 
is to ignore the slide altogether and send it to the 
output hopper . 

The HER2 -positive application (1) finds the 

15 specimen, (2) goes to five locations within the 
specimen, (3) stores the image from each location, 
(4) completes color space transformations to obtain 
the intensity of stain at all five locations, and (5) 
stores the average value of intensity in the database 

20 (DB) . This value is used as the normalized 

expression value for the HER2 quantitative analysis. 
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This value is also used to determine the A between 
the positive and negative control slides and verify- 
that the A is large enough. 

The HER2 -negative application (1) finds the 
5 specimen, (2) goes to five locations within the 
specimen, (3) stores the image from each location, 
(4) completes color space transformations to obtain 
the intensity of stain at all five locations, and (5) 
stores the average value of intensity in the 

10 database. This value is used to determine the A 

between the positive and negative control slides and 
verify that the A is large enough. 

The HER2 application (1) finds the specimen, (2) 
scans the slide at lOx and constructs a histological 

15 reconstruction object, and (3) stores the object in 
the database. Before a slide is analyzed, the 
instrument checks to see if the hard drive space has 
met a certain capacity for archiving data. If so, 
the user is prompted to archive before continuing 

20 instrument analysis. Compression may need to take 
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place prior to the object being stored within the 
database . 

After completion of running all the slides 
within the worklist, the histological reconstruction 
5 objects are analyzed by the pathologist or physician. 

The Slide Display Interface displays the list of 
slides that were analyzed, or the list of patient 
cases. Within tissue analysis, there are always at 
least two slides per patient case that are analyzed 
10 by the instrument. The set of patient slides is a 
i c „. logical unit. Therefore, a pathologist or physician 
k may choose the patient case by the patient ID or the 
accession number. 

Within the User Interface that displays the list 
15 of patient cases to be reviewed/ analyzed, once the 
pathologist chooses the patient case, H/E image is 
always displayed first. Using the H/E image, the 
pathologist determines whether or not the cancer is 
invasive and whether or not analysis of the IHC slide 
20 is necessary. 
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Upon patient case selection, the low power H/E 
image is displayed. This image can be reduced to 
enable the whole tissue cross -section to be displayed 
within the system's monitor (resolution at least 1024 
5 x 760) . 

The pathologist can have the features available 
to choose more than one patient case to review and 
then traverse from one case to the next with the 
following features. For each patient case, the 

10 reduced low power H/E image always appears first. 

While viewing the reduced low power H/E image 
the pathologist has the following capabilities: (1) 
to zoom into an area of the low power H/E image for 
higher magnification; (2) to choose to go to the 

15 reduced low power immunohistochemistry image for the 
current patient case; (3) to select an area to 
incorporate within the patient report (Marking an 
image section) ; (4) to choose to go to the next 
patient case (if selected in previous user 

20 interface) ; and (5) to choose to go to the previous 
patient case. 
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While viewing the zoomed H/E image the 
pathologist has the following capabilities: (1) to 
go back to the reduced low power H/E image; (2) to 
choose to go to the reduced low power 
5 immunohistochemistry image for the current patient 
case; (3) to select an area to incorporate within the 
patient report (Marking an image section) ; (4) to 
choose to go to the next patient case (if selected in 
previous user interface) ; and (5) to choose to go to 

10 the previous patient case. 

While viewing the reduced low power 
immunohistochemistry image the pathologist has the 
following capabilities: (1) to zoom into an area of 
the low power immunohistochemistry image for higher 

15 magnification; (2) to choose to go back to the 

reduced H/E image for the current patient case; (3) 
to select an area of the reduced immunohistochemistry 
image and query for quantitative results; (4) to 
select an area to incorporate within the patient 

20 report (Marking an image section) ; (5) to select a 

quantitative result to incorporate within the patient 
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report (Save the quantitative result) ; (6) to choose 
to go to the next patient case (if selected in 
previous user interface) ; and (7) to choose to go to 
the previous patient case (if applicable) . 
5 While viewing the zoomed immunohistochemistry 

image the pathologist has the following capabilities : 

(1) to go back to the reduced low power IHC image; 

(2) to choose to go back to the reduced H/E image for 
the current patient case; (3) to select an area of 

10 the zoomed immunohistochemistry image and query for 
quantitative results; (4) to select an area to . 
incorporate within the patient report (Marking an\ 
image section); (5) to select a quantitative result 
to incorporate within the patient report (Save the 

15 quantitative result) ; (6) to choose to go to the next 
patient case (if selected in previous user 
interface) ; and (7) to choose to go to the previous 
patient case (if applicable) . 

After completion of the pathologist analysis and 

20 review of all the patient cases, the reports can be 
generated. A User Interface lists all the patient 
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cases that have already been reviewed and analyzed by 
the pathologist. The pathologist can choose one or 
more of the listed patient cases to view the reports 
electronically. The HER2 / neu report for a patient 
5 case can contain the following information: (1) 
Patient Information and Demographics as entered 
within the Worklist Information Entry; (2) Referring 
Physician and Hospital as entered within the Worklist 
Information Entry; (3) Pathologist and Laboratory 

10 Information; (4) Sections of the H/E image marked 
during the review/analysis for report inclusion; (5) 
sections of the immunohistochemistry image marked 
during the review/analysis, for report inclusion; 
(5) quantitative analysis saved during the 

15 review/analysis for report inclusion; and (6) 

HER2 / neu application and scoring analysis reference 
materials. 

The interface that displays the patient reports 
online contains the following functionality: (1) 
20 Print; (2) Send To, allows for (linked to e-mail, 
when available, and prompting the user for the 
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address of where to send the report) ; (3) Next 
Patient Case Report (if more than one case was 
selected from the previous interface) ; (4) Previous 
Patient Case Report (if applicable) ; and 
5 (5) Save/Apply (if the additional comments area is 
modified) . Network connectivity is site specific. 
Site specificity is desired for sending reports via 
the Internet without having to transport the report 
over to another host to send it. Direct sending 

10 capabilities are preferred. 

The instrument can have at least two user modes. 
One user mode is a locked protocol consistent with 
FDA regulations. A second user mode is modifiable by 
the user. For the second user mode, access to the 

15 following parameters of the application include: (1) 
definition of the scan area, center, width, and 
height; (2) toggle switch for whether or not the Find 
phase is used; (3) Focus Type; (4) Focus Threshold; 
and (5) Focus Inset. 
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Automated System 

Referring now to the figures, an apparatus for 
automated cell analysis of biological specimens is 
generally indicated by reference numeral 10 as shown 
5 in perspective view in Fig. 1 and in block diagram 
form in Fig. 2. The apparatus 10 comprises a 
microscope subsystem 32 housed in a housing 12. The 
housing 12 includes a slide carrier input hopper 16 
and a slide carrier output hopper 18 . A door 14 in 

10 the housing 12 secures the microscope subsystem from 
the external environment. A computer subsystem :v 
comprises a computer 22 having a system processor 23, 
an image processor 25 and a communications modem 29. 
The computer subsystem further includes a computer 

15 monitor 26 and an image monitor 27 and other external 
peripherals including storage device 21 , pointing 
device 30, keyboard 28 and color printer 35. An 
external power supply 24 is also shown for powering 
the system. Viewing oculars 20 of the microscope 

20 subsystem project from the housing 12 for operator 
viewing. The apparatus 10 further includes a CCD 
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camera 42 for acquiring images through the microscope 
subsystem 32. A microscope controller 31 under the 
control of system processor 23 controls a number of 
microscope- subsystem functions described further in 
5 detail. An automatic slide feed mechanism in 

conjunction with X-Y stage 38 provide automatic slide 
handling in the apparatus 10. An illumination light 
source 48 projects light onto the X-Y stage 38 which 
is subsequently imaged through the microscope 

10 subsystem 32 and acquired through CCD camera 42 for 
processing in the image processor 25. A Z stage or 
focus stage 46 under control of the microscope 
controller 31 provides displacement of the microscope 
subsystem in the Z plane for focusing. The 

15 microscope subsystem 32 further includes a motorized 
objective turret 44 for selection of objectives. 

The purpose of the apparatus 10 is for the 
unattended automatic scanning of prepared microscope 
slides for the detection and counting of candidate 

20 objects of interest such as normal and abnormal 

cells, e.g., tumor cells. The preferred embodiment 
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may be utilized for rare event detection in which 
there may be only one candidate object of interest 
per several hundred thousand normal cells, e.g., one 
to five candidate objects of interest per 2 square 
5 centimeter area of the slide. The apparatus 10 

automatically locates and counts candidate objects of 
interest and estimates normal cells present in a 
biological specimen on the basis of color, size and 
shape characteristics. A number of stains are used 

10 to preferentially stain candidate objects of interest 
and normal cells different colors so that such cells 
can be distinguished from each other. 
- • • As noted in the background of the invention, a 
biological specimen may be prepared with a reagent to 

15 obtain a colored insoluble precipitate. The 

apparatus of the present invention is used to detect 
this precipitate as a candidate object of interest. 
During operation of the apparatus 10, a pathologist 
or laboratory technician mounts prepared slides onto 

20 slide carriers. A slide carrier 60 is illustrated in 
Fig. 8 and will be described further below. Each 
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slide carrier holds up to 4 slides. Up to 25 slide 
carriers are then loaded into input hopper 16 . The 
operator can specify the size, shape and location of 
the area to be scanned or alternatively, the system 
5 can automatically locate this area. The operator 
then commands the system to begin automated scanning 
of the slides through a graphical user 25 interface. 
Unattended scanning begins with the automatic loading 
of the first carrier and slide onto the precision 

10 motorized X-Y stage 38. A bar code label affixed to 
the slide is read by a bar code reader .3 3 during this 
loading operation. Each slide is then scanned at a 
user selected low microscope magnification, for 
example, lOx, to identify candidate objects of 

15 interest based on their color, size, and shape 
characteristics . The X-Y locations of candidate 
cells are stored until scanning is completed. 

After the low magnification scanning is 
completed, the apparatus automatically returns to 

20 each candidate object of interest, reimages and 

refocuses at a higher magnification such as 40x and 
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performs further analysis to confirm the object 
candidate. The apparatus stores an image of the 
object for later review by a pathologist. All 
results and images can be stored to a storage device 
5 21 such as a removable hard drive or DAT tape or 
transmitted to a remote site for review or storage. 
The stored images for each slide can be viewed in a 
mosaic of images for further review. In addition, 
the pathologist or operator can also directly view a 

10 detected object through the microscope using the 
included oculars '20 or on image monitor 27. 

Having described the overall operation of the 
apparatus 10 from a 'high level, the further details 
of the apparatus will now be described. Referring to 

15 Fig. 3, the microscope controller 31 is shown in more 
detail. The microscope controller 31 includes a 
number of subsystems connected through a system bus. 
A system processor 102 controls these subsystems and 
is controlled by the apparatus system processor 23 

20 through an RS 232 controller 110. The system 
processor 102 controls a set of motor - control 
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subsystems 114 through 124 which control the input 
and output feeder, the motorized turret 44, the X-Y 
stage 38, and the Z stage 46 (Fig. 2) . A histogram 
processor 108 receives input from CCD camera 42 for 
5 computing variance data during the focusing operation 
described further herein. The system processor 102 
further controls an illumination controller 106 for 
control of substage illumination 48. The light 
output from the halogen light bulb which supplies 

10 illumination for the system can vary over time due to 
bulb aging, changes in optical alignment, and other . / 
factors. In addition, slides which have been Wover 
stained" can reduce the camera exposure to an 
unacceptable level. In order to compensate for these 

15 effects, the illumination controller 106 is included. 
This controller is used in conjunction with light 
control software to compensate for the variations in 
light level. The light control software samples the 
output from the camera at intervals (such as between 

20 loading of slide carriers) , and commands the 

controller to adjust the light level to the desired 
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levels. In this way, light control is automatic and 
transparent to the user and adds no substantial 
additional time to system operation. The system 
processor 23 may be any processor capable of 
5 performing the operations of the system, for example 
dual parallel Intel Pentium 90 MHZ devices, a T.I. 
C80 processor for computation and processors capable 
of 8 to 32 bit operation on an NT system. The image 
processor 25 is preferably a Matrox Imaging Series 

10 640 model. The microscope controller system 

processor 102 is an Advanced Micro Devices AMD29K 
device . m; * 

Figs. 4 and 5 show further detail of the 
apparatus 10 is shown. Fig. 4 shows a plan view of 

15 the apparatus 10 with the housing 12 removed. A 

portion of the automatic slide feed mechanism 37 is 
shown to the left of the microscope subsystem 32 and 
includes slide carrier unloading assembly 34 and 
unloading platform 36 which in conjunction with slide 

20 carrier output hopper 18 function to receive slide 
carriers which have been analyzed. Vibration 
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isolation mounts 40, shown in further detail in Fig. 
5, are provided to isolate the microscope subsystem 
32 from mechanical shock and vibration that can occur 
in a typical laboratory environment. In addition to 
5 external sources of vibration, the high-speed 

operation of the X-Y stage 38 can induce vibration 
into the microscope subsystem 32. Such sources of 
vibration can be isolated from the electro-optical 
subsystems to avoid any undesirable effects on image 

10 quality. The isolation mounts 40 comprise a spring 
40a^and piston 40b submerged in a high viscosity 
silicon gel which is enclosed in an elastomer 
membrane bonded to a casing to achieve damping 
factors on the order of 17 to 20%. 

15 The automatic slide-handling feature of the 

present invention will now be described. The 
automated slide handling subsystem operates on a 
single slide carrier at a time. A slide carrier 60 
is shown in Figs. 6a and 6b which provide a top view 

20 and a bottom view respectively. The slide carrier 60 
includes up to four slides 70 mounted with adhesive 
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tape 62. The carrier 60 includes ears 64 for hanging 
the carrier in the output hopper 18. An undercut 66 
and pitch rack 68 are formed at the top edge of the 
slide carrier 60 for mechanical handling of the slide 
5 carrier. A keyway cutout 65 is formed in one side of 
the carrier 60 to facilitate carrier alignment. A 
prepared slide 72 mounted on the slide carrier 60 
includes a sample area 72a and a bar code label area 
72b. Fig. 7a provides a top view of the slide 

10 handling subsystem which comprises a slide input 
module 15, a slide output module 17 and X-Y stage 
drive belt 50. Fig. 7b provides a partial cross- 
sectional view taken along line A-A of Fig. 7a. The 
slide input module 15 comprises a slide carrier input 

15 hopper 16, loading platform 52 and slide carrier 

loading subassembly 54. The input hopper 16 receives 
a series of slide carriers 60 (Figs. 6a and 6b) in a 
stack on loading platform 52. A guide key 57 
protrudes from a side of the input hopper 16 to which 

20 the keyway cutout 65 (Fig. 6a) of the carrier is fit 
to achieve proper 15 alignment. The input module 15 
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further includes a revolving indexing cam 56 and a 
switch 90 mounted in the loading platform 52, the 
operation of which is described further below. The 
carrier loading subassembly 54 comprises an infeed 
5 drive belt 59 driven by a motor 86. The infeed drive 
belt 59 includes a pusher tab 58 for pushing the 
slide carrier horizontally toward the 20 X-Y stage 38 
when the belt is driven. A homing switch 95 senses 
the pusher tab 58 during a revolution of the belt 59. 

10 Referring specifically to Fig. 7a, the X-Y stage 

38 is shown with^x position and y position motors 96 
and 97 respectively which are controlled by the 
microscope controller. 31 (Fig. 3) and are not 
considered part of the slide handling subsystem. The 

15 X-Y stage 38 further includes an aperture 55 for 

allowing illumination to reach the slide carrier. A 
switch 91 is mounted adjacent the aperture 55 for 
sensing contact with the carrier and thereupon 
activating a motor 87 to drive stage drive belt 50 

20 (Fig. 7b) . The drive belt 50 is a double- sided 
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timing belt having teeth for engaging pitch rack 68 
of the carrier 60 (Fig. 6b) . 

The slide output module 17 includes slide 
carrier output hopper 18, unloading platform 36 and 
5 slide carrier unloading subassembly 34. The 

unloading subassembly 34 comprises a motor 89 for 
rotating the unloading platform 36 about shaft 98 
during an unloading operation described further 
below. An out feed gear 93 driven by motor 88 
10 rotatably engages the pitch rack 68 of the carrier 60 
;(Fig. 6b) to transport the carrier to a rest position 
'against switch 92. A springloaded hold-down 
^mechanism holds the carrier in place on the unloading 
platform 36, 

15 The slide handling operation will now be 

described. Referring to Fig. 8, a series of slide 
carriers 60 are shown stacked in input hopper 16 with 
the top edges 60a aligned. As the slide handling 
operation begins, the indexing cam 56 driven by motor 

20 85 advances one revolution to allow only one slide 
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carrier to drop to the bottom of the hopper 16 and 
onto the loading platform 52 . 

Figs. 8a-8d show the cam action in more detail. 
The cam 56 includes a hub 56a to which are mounted 
5 upper and lower leaves 56b and 56c respectively. The 
leaves 56b, 56c are semicircular projections 
oppositely positioned and spaced apart vertically. 
In a first position shown in Fig. 8a, the upper leaf 
56b supports the bottom carrier at the undercut 

10 portion 66. At a 20 position of the cam 56 rotated 
180° , shown in Fig. 8b, the upper leaf 56b no longer 
supports the carrier and instead the carrier has 
dropped slightly and is supported by the lower leaf 
56c. Fig. 8c shows the position of the cam 56 

15 rotated 270° wherein the upper leaf 56b has rotated 
sufficiently to begin to engage the undercut 66 of 
the next slide carrier while the opposite facing 
lower leaf 56c still supports the bottom carrier. 
After a full rotation of 360° as shown in Fig. 8d,the 

20 lower leaf 56c has rotated opposite the carrier stack 
and no longer supports the bottom carrier which now 
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rests on the loading platform 52 . At the same 
position, the upper leaf 56b supports the next 
carrier for repeating the cycle. 

Referring again to Figs. 7a and 7b, when the 
5 carrier drops to the loading platform 52, the contact 
closes switch 90 which activates motors 86 and 87. 
Motor 86 drives the infeed drive belt 59 until the 
pusher tab 58 makes contact with the carrier and 
pushes the carrier onto the X-Y stage drive belt 50. 

10 The stage drive belt 50 advances the carrier until 
contact is made; with, switch 91, the closing of which 
begins the slide scanning process described further 
herein. Upon , completion of the scanning process, the 
X-Y stage 38 moves to an unload position and motors 

15 87 and 88 are activated to transport the carrier to 
the unloading platform 36 using stage drive belt 50. 
As noted, motor 88 drives out feed gear 93 to engage 
the carrier pitch rack 68 of the carrier 60 (Fig. 6b) 
until switch 92 is contacted. Closing switch 92 

20 activates motor 89 to rotate the unloading platform 
36. 
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The unloading operation is shown in more detail 
in end views of the output module 17 (Figs. 9a-9d) . 
In Fig. 9a, the unloading platform 36 is shown in a 
horizontal position supporting a slide carrier 60. 
5 The hold-down mechanism 94 secures the carrier 60 at 
one end. Fig. 9b shows the output module 17 after 
motor 89 has rotated the unloading platform 36 to a 
vertical position, at which point the spring loaded 
hold-down mechanism 94 releases the slide carrier 60 

10 into the output hopper 18. The carrier 60 is 

supported in the output hopper 18 by means of ears * 64 
(Figs. 6a and 6b). Fig. 9c shows the unloading . 
platform 36 being rotated back towards the 20 
horizontal position. As the platform 36 rotates 

15 upward, it contacts the deposited carrier 60 and the 
upward movement pushes the carrier toward the front 
of the output hopper 18. Fig. 9d shows the unloading 
platform 36 at its original horizontal position after 
having output a series of slide carriers 60 to the 

20 output hopper 18. 
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Having described the overall system and the 
automated slide handling feature, the aspects of the 
apparatus 10 relating to scanning, focusing and image 
processing will now be described in further detail. 
5 In some cases, an operator will know ahead of 

time where the scan area of interest is on the slide. 
Conventional preparation of slides for examination 
provides repeatable and known placement of the sample 
on the slide. The operator can therefore instruct 

10 the system to always scan the same area at the same 
location of every slide which is: prepared in this 
fashion. But there are other times in which the area 
of interest is not known, for example, where slides 
are prepared manually with a known smear technique. 

15 One feature of the invention automatically determines 
the scan area using a texture analysis process. Fig. 
10 is a flow diagram that describes the processing 
associated with the automatic location of a scan 
area. As shown in this figure, the basic method is 

20 to pre-scan the entire slide area to determine 
texture features that indicate the presence of a 
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smear and to discriminate these areas from dirt and 
other artifacts. 

At each location of this raster scan, an image 
such as in Fig. 12 is acquired and analyzed for 
5 texture information at steps 204 and 206. Since it 
is desired to locate the edges of the smear sample 
within a given image, texture analyses are conducted 
over areas called windows 78, which are smaller than 
the entire image as shown in Fig. 12. The process 

10 iterates the scan across the slide at steps 
208,210,212 and 214. 

; :In the interest of speed, the texture analysis 
process is performed at a lower magnification, 
preferably at a 4x objective. One reason to operate 

15 at low magnification is to image the largest slide 

area at any one time. Since cells do not yet need to 
be resolved at this stage of the overall image 
analysis, the 4x magnification is preferred. On a 
typical slide, as shown in Fig. 11, a portion 72b of 

20 the end of the slide 72 is reserved for labeling with 
identification information. Excepting this label 
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area, the entire slide is scanned in a raster scan 
fashion 76 to yield a number of adjacent images. 
Texture values for each window include the pixel 
variance over a window, the difference between the 
5 largest and smallest pixel value within a window, and 
other indicators. The presence of a smear raises the 
texture values compared with a blank area. 

One problem with a smear from the standpoint of 
determining its location is its non-uniform thickness 

10 and texture. For example, the smear is likely to be 
relatively thin at the edges and thicker towards the 
middle due to the nature of the smearing process . To 
accommodate for the non-uniformity, texture analysis 
provides a texture value for each analyzed area. The 

15 texture value tends to gradually rise as the scan 
proceeds across a smear from a thin area to a thick 
area, reaches a peak, and then falls off again to a 
lower value as a thin area at the edge is reached. 
The problem is then to decide from the series of 

20 texture values the beginning and ending, or the 

edges, of the smear. The texture values are fit to a 
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square wave waveform since the texture data does not 
have sharp beginnings and endings. 

After conducting this scanning and texture 
evaluation operation, one must determine which areas 
5 of elevated texture values represent the desired 
smear 74, and which represent undesired artifacts. 
This is accomplished by fitting a step function, on a 
line-by-line basis to the texture values in step 216. 
This function, which resembles a single square wave 

10 across the smear with a beginning at one edge, and 
end at the other edge, and an amplitude provides the 
means for discrimination. The amplitude of the best- 
fit step function is utilized to determine whether 
smear or dirt is present since relatively high values 

15 indicate smear. If it is decided that smear is 

present, the beginning and ending coordinates of this 
pattern are noted until all lines have been 
processed, and the smear sample area defined at 218. 
After an initial focusing operation described 

20 further herein, the scan area of interest is scanned 
to acquire images for image analysis. The preferred 
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method of operation is to initially perform a 
complete scan of the slide at low magnification to 
identify and locate candidate objects of interest, 
followed by further image analysis of the candidate 
5 objects of interest at high magnification in order to 
confirm the objects as cells. An alternate method of 
operation is to perform high magnification image 
analysis of each candidate object of interest 
immediately after the object has been identified at 

10 low magnification. The low magnification scanning 
then resumes, searching for additional candidate 
objects of interest. Since it takes on the order of 
a few seconds to change objectives, this alternate 
method of operation would take longer to complete. 

15 The operator can pre -select a magnification 

level to be used for the scanning operation. A low 
magnification using a lOx objective is preferred for 
the scanning operation since a larger area can be 
initially analyzed for each acquired scan image. The 

20 overall detection process for a cell includes a 

combination of decisions made at both low (lOx) and 
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high magnification (40x) levels. Decision making at 
the lOx magnification level is broader in scope, 
i.e., objects that loosely fit the relevant color, 
size and shape characteristics are identified at the 
5 lOx level. 

Analysis at the 40x magnification level then 
proceeds to refine the dec is ion -making and confirm 
objects as likely cells or candidate objects of 
interest. For example, at the 40x level it is not 

10 uncommon to find that some objects that were 

identified at lOx are artifacts which the analysis, 
process will then reject. In addition, closely 
packed objects of interest appearing at lOx are 
separated at the 40x level. 

15 In a situation where a cell straddles or 

overlaps adjacent image fields, image analysis of the 
individual adjacent image fields could result in the 
cell being rejected or undetected. To avoid missing 
such cells, the scanning operation compensates by 

20 overlapping adjacent image fields in both the x and y 
directions. An overlap amount greater than half the 
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diameter of an average cell is preferred. In the 
preferred embodiment, the overlap is specified as a 
percentage of the image field in the x and y 
directions . 

5 The time to complete an image analysis can vary 

depending upon the size of the scan area and the 
number of candidate cells, or objects of interest 
identified. For one example, in the preferred 
embodiment, a complete image analysis of a scan area 

10 of two square centimeters in which 50 objects of 

interest are confirmed can be performed in about 12 
to ;15 minutes. This example includes not only 
focusing, scanning, and image analysis, but also the 
saving of 4 Ox images as a mosaic on hard drive 21 

15 (Fig. 2) . Consider the utility of the present 

invention in a "rare event" application where there 
may be one, two or a very small number of cells of 
interest located somewhere on the slide. To 
illustrate the nature of the problem by analogy, if 

20 one were to scale a slide to the size of a football 
field, a tumor cell, for example, would be about the 
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size of a bottle cap. The problem is then to rapidly 
search the football field and find the very small 
number of bottle caps and have a high certainty that 
none have been missed. 
5 However the scan area is defined, an initial 

focusing operation must be performed on each slide 
prior to scanning. This is required since slides 
differ, in general, in their placement in a carrier. 
These differences include slight (but significant) 

10 variations of tilt of the slide in its carrier. 
Since each slide must remain in focus during 
scanning, the degree of tilt of each slide must be 
determined. This is accomplished with an initial 
focusing operation that determines the exact degree 

15 of tilt, so that focus can be maintained 
automatically during scanning. 

The initial focusing operation and other 
focusing operations to be described later utilize a 
focusing method based on processing of images 

20 acquired by the system. This method was chosen for 
its simplicity over other methods including use of IR 
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beams reflected from the slide surface and use of 
mechanical gauges . These other methods also would 
not function properly when the specimen is protected 
with a coverglass. The preferred method results in 
5 lower system cost and improved reliability since no 
additional parts need be included to perform 
focusing. Fig. 13A provides a flow diagram 
describing the "focus point" procedure. The basic 
method relies on the fact that the pixel value 

10 variance (or standard deviation) taken about the 
pixel value mean is maximum at best focus. A 
"brute -force" method icould simply step through 
focus, using the computer controlled Z or focus 
stage, calculate the pixel variance at each step, and 

15 return to the focus position providing the maximum 
variance. Such a method would be too time consuming. 
Therefore, additional features were added as shown in 
Fig. 13A. 

These features include the determination of 
20 pixel variance at a relatively coarse number of focal 
positions, and then the fitting of a curve to the 
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data to provide a faster means of determining optimal 
focus. This basic process is applied in two steps, 
coarse and fine. During the coarse step at 220-230, 
the Z stage is stepped over a user-specified range of 
5 focus positions, with step sizes that are also user- 
specified. It has been found that for coarse 
focusing, these data are a close fit to a Gaussian 
function. Therefore, this initial set of variance 
versus focus position data are least-squares fit to a 
10 Gaussian function at 228. The location of the peak 
rpf this Gaussian curve determines the initial or 
coarse estimate of focus position for input to step 
c23.2 

Following this, a second stepping operation 232- 
15 242 is performed utilizing smaller steps over a 
smaller focus range centered on the coarse focus 
position. Experience indicates that data taken over 
this smaller range are generally best fit by a second 
order polynomial. Once this least squares fit is 
20 performed at 240, the peak of the second order curve 
provides the fine focus position at 244. 
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Fig. 14 illustrates a procedure for how this 
focusing method is utilized to determine the 
orientation of a slide in its carrier. As shown, 
focus positions are determined, as described above, 
5 for a 3 x 3 grid of points centered on the scan area 
at 264. Should one or more of these points lie 
outside the scan area, the method senses at 266 this 
by virtue of low values of pixel variance. In this 
case, additional points are selected closer to the 

10 center of the scan area. Fig. 15 shows the initial 
array of points 80 and new point 82 selected closer 
to the center. Once this array of. focus positions is 
determined at 268, a least squares. plane is fit to 
this data at 270. Focus points lying too far above 

15 or below this best-fit plane are discarded at 272 

(such as can occur from a dirty cover glass over the 
scan area), and the data is then refit. This plane 
at 274 then provides the desired Z position 
information for maintaining focus during scanning. 

20 After determination of the best -fit focus plane, 

the scan area is scanned in an X raster scan over the 



WO 01/04828 




PCT/US00/19046 



76 



scan area as described earlier. During scanning, the 
X stage is positioned to the starting point of the 
scan area, the focus (Z) stage is positioned to the 
best fit focus plane, an image is acquired and 
5 processed as described herein, and this process is 
repeated for all points over the scan area. In this 
way f focus is maintained automatically without the 
need for time-consuming refocusing at points during 
scanning. Prior to confirmation of cell objects at a 

10 4 Ox or 6 Ox level, a refocusing operation is conducted 
since the use of; this higher magnification requires 
more precise focus than the best-fit plane provides. 
Fig. 16 provides ithe flow diagram for this process. 
As may be seen, this process is similar to the fine 

15 focus method described earlier in that the object is 
to maximize the image pixel variance. This is 
accomplished by stepping through a range of focus 
positions with the Z stage at 276, 278, calculating 
the image variance at each position at 278, fitting a 

20 second order polynomial to these data at 282, and 
calculating the peak of this curve to yield an 
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estimate of the best focus position at 284, 286. 
This final focusing step differs from previous ones 
in that the focus range and focus step sizes are 
smaller since this magnification requires focus 
5 settings to within 0.5 micron or better. It should 
be noted that for some combinations of cell staining 
characteristics, improved focus can be obtained by 
numerically selecting the focus position that 
provides the largest variance, as opposed to 

10 selecting the peak of the polynomial. In such cases, 
the polynomial is used to provide an estimate of best 
.: focus, and a final step selects the actual Z position 
; giving highest pixel variance. It should also be 
noted that if at any time during the focusing process 

15 at 40x or 60x the parameters indicate that the focus 
position is inadequate, the system automatically 
reverts to a coarse focusing process as described 
above with reference to Fig. 13A. This ensures that 
variations in specimen thickness can be accommodated 

20 in an expeditious manner. For some biological 

specimens and stains, the focusing methods discussed 
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above do not provide optimal focused results. For 
example, certain white blood cells known as 
neutrophils may be stained with Fast Red, a commonly 
known stain, to identify alkaline phosphatase in the 
5 cytoplasm of the cells. To further identify these 
cells and the material within them, the specimen may 
be counterstained with hemotoxylin to identify the 
nucleus of the cells. In cells so treated, the 
cytoplasm bearing alkaline phosphatase becomes a 

10 shade of red proportionate to the amount of alkaline 
phosphatase in the cytoplasm and the nucleus becomes 
blue. However, where the cytoplasm and nucleus 
overlap, the cell appears purple. These color 
combinations appear to preclude the finding of a 

15 focused Z position using the focus processes 
discussed above. 

In an effort to find a best focal position at 
high magnification, a focus method, such as the one 
shown in Fig. 13B, may be used. That method begins 

20 by selecting a pixel near the center of a candidate 
object of interest (Block 248) and defining a region 
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of interest centered about the selected pixel (Block 
250) . Preferably, the width of the region of 
interest is a number of columns which is a power of 
2 . This width preference arises from subsequent 
5 processing of the region of interest preferably using 
a one dimensional Fast Fourier Transform (FFT) 
technique. As is well known within, the art, 
processing columns of pixel values using the FFT 
technique is facilitated by making the number of 

10 columns to be processed a power of two. While the 
height :of the. region of interest is also a power of 
two in -the -preferred embodiment, it need not be 
unless : a two dimensional FFT technique is used to 
process the region of interest. 

15 After the region of interest is selected, the 

columns of pixel values are processed using the 
preferred one-dimensional FFT to determine a spectra 
of frequency components for the region of interest 
(Block 252) . The frequency spectra ranges from DC to 

20 some highest frequency component. For each frequency 
component, a complex magnitude is computed. 
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Preferably, the complex magnitudes for the frequency 
components which range from approximately 25% of the 
highest component to approximately 75% of the highest 
component are squared and summed to determine the 
5 total power for the region of interest (Block 254) . 
Alternatively, the region of interest may be 
processed with a smoothing window, such as a Harming 
window, to reduce the spurious high frequency 
components generated by the FFT processing of the 

10 pixel values in the region of interest. Such 

preprocessing of the region of interest permits all 
complex magnitude over the complete frequency range 
to be squared and summed. After the power for a 
region has been computed and stored (Block 256) , a 

15 new focal position is selected, focus adjusted 

(Blocks 258, 260), and the process repeated. After 
each focal position has been evaluated, the one 
having the greatest power factor is selected as the 
one best in focus (Block 262) . 

20 The following describes the image processing 

methods which are utilized to decide whether a 
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candidate object of interest such as a stained tumor 
cell is present in a given image, or field, during 
the scanning process. Candidate objects of interest 
which are detected during scanning are reimaged at 
5 higher (40x or 60x) magnification, the decision 
confirmed, and a region of interest for this cell 
saved for later review by the pathologist. The image 
processing includes color space conversion, low pass 
filtering, background suppression, artifact 

10 suppression, morphological processing, and blob 

analysis. One or more of these steps can optionally 
be eliminated. The operator is provided with an 
option to configure the system to perform any or all 
of these steps and whether to perform certain steps 

15 more than once or several times in a row. It should 
also be noted that the sequence of steps may be 
varied and thereby optimized for specific reagents or 
reagent combinations; however, the sequence described 
herein is preferred. It should be noted that the 

20 image processing steps of low pass filtering, 

thresholding, morphological processing, and blob 
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analysis are generally known image processing 
building blocks. 

An overview of the preferred process is shown in 
Fig. 17A. The preferred process for identifying and 
5 locating candidate objects of interest in a stained 
biological specimen on a slide begins with an 
acquisition of images obtained by scanning the slide 
at low magnification (Block 288) . Each image is then 
converted from a first color space to a second color 

10 space (Block 290) and the color converted image is 

low pass filtered (Block 292). The pixels of the low 
pass*: filtered image are then compared to a threshold 
(Block 294):: and, preferably, those pixels having a 
value equal to or greater than the threshold are 

15 identified as candidate object of interest pixels and 
those less than the threshold are determined to be 
artifact or background pixels. The candidate object 
of interest pixels are then morphologically processed 
to identify groups of candidate object of interest 

20 pixels as candidate objects of interest (Block 296) . 
These candidate objects of interest are then compared 
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to blob analysis parameters (Block 2 98) to further 
differentiate candidate objects of interest from 
objects which do not conform to the blob analysis 
parameters and, thus, do not warrant further 
5 processing. The location of the candidate objects of 
interest may be stored prior to confirmation at high 
magnification. The process continues by determining 
whether the candidate objects of interest have been 
confirmed (Block 300) . If they have not been 

10 confirmed, the optical system is set to high 

magnification (Block 302) and images of the slide at 
the locations corresponding to the candidate objects 
of interest identified in the low magnification 
images are acquired (Block 288) . These images are 

15 then color converted (Block 290) , low pass filtered 
(Block 292), compared to a threshold (Block 294), 
morphologically processed (Block 296) , and compared 
to blob analysis parameters (Block 298) to confirm 
which candidate objects of interest located from the 

20 low magnification images are objects of interest. 
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The coordinates of the objects of interest are then 
stored for future reference (Block 303) . 

In general, the candidate objects of interest, 
such as tumor cells, are detected based on a 
5 combination of characteristics, including size, 

shape, and color. The chain of dec is ion -making based 
on these characteristics preferably begins with a 
color space conversion process. The CCD camera 
coupled to the microscope subsystem outputs a color 
10 image comprising a matrix of 640 x 480 pixels. Each 
pixel comprises red, green, and blue (ROB) signal 
values. 

It is desirable to transform the matrix of RGB 
values to a different color space because the 

15 difference between candidate objects of interest and 
their background, such as tumor and normal cells, may 
be determined from their respective colors. 
Specimens are generally stained with one or more 
industry standard stains (e.g., DAB, New Fuchsin, 

20 AEC) which are 11 reddish" in color. Candidate 

objects of interest retain more of the stain and thus 
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appear red while normal cells remain unstained. The 
specimens may also be counterstained with hematoxalin 
so the nuclei of normal cells or cells not containing 
an object of interest appear blue. In addition to 
5 these objects, dirt and debris can appear as black, 
gray, or can also be lightly stained red or blue 
depending on the staining procedures utilized. The 
residual plasma or other fluids also present on a 
smear may also possess some color. 

10 In the color conversion operation, a ratio of 

two of the RGB signal values is formed to provide ai- 
means for discriminating color information. With 
-..•three signal values for each pixel, nine different 
ratios can be formed: R1R, R/G, RUB, G/G, G/B, G/R, 

15 B/B, B/G, B/R. The optimal ratio to select depends 
upon the range of color information expected in the 
slide specimen. As noted above, typical stains used 
for detecting candidate objects of interest such as 
tumor cells are predominantly red, as opposed to 

20 predominantly green or blue. Thus, the pixels of a 
cell of interest which has been stained contain a red 
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component which is larger than either the green or 
blue components. A ratio of red divided by blue 
(R/B) provides a value which is greater than one for 
tumor cells but is approximately one for any clear or 
5 white areas on the slide. Since the remaining cells, 
i.e., normal cells, typically are stained blue, the 
R/B ratio for pixels of these latter cells yields 
values of less than one. The R/B ratio is preferred 
for clearly separating the color information typical 

10 in these applications . 

Fig. 17B illustrates the flow, diagram by which 
this conversion is performed. In the interest of 
processing speed, the conversion is implemented with 
a look up table. The use of a look up table for 

15 color conversion accomplishes three functions: (1) 
performing a division operation; (2) scaling the 
result for processing as an image having pixel values 
ranging from 0 to 255; and (3) defining objects which 
have low pixel values in each color band (R,G,B) as 

20 "black" to avoid infinite ratios (i.e., dividing by 
zero) . These "black" objects are typically 
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staining artifacts or can be edges of bubbles caused 
by pasting a coverglass over the specimen. 

Once the look up table is built at 304 for the 
specific color ratio (i.e., choices of tumor and 
5 nucleated cell stains) , each pixel in the original 
RGB image is converted at 308 to produce the output. 
Since it is of interest to separate the red stained 
tumor cells from blue stained normal ones, the ratio 
of color values is then scaled by a user specified 

10 factor. As an example, for a factor of 128 and the 
ratio. of (red pixel value) /(blue pixel value), clear 
areas on the slide would have a ratio of 1 scaled by 
128 for a final X value of 128. Pixels which lie in 
red stained tumor 

15 cells would have X value greater than 128, while blue 
stained nuclei of normal cells would have value less 
than 128. In this way, the desired objects of 
interest can be numerically discriminated. The 
resulting 640 x 480 pixel matrix, referred to as the 

20 X- image, is a gray scale image having values ranging 
from 0 to 255. 
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Other methods exist for discriminating color 



information. One classical method converts the RGB 
color information into another color space, such as 
HSI (hue, saturation, intensity) space. In such a 
5 space, distinctly different hues such as red, blue, 
green, yellow, may be readily separated. In 
addition, relatively lightly stained objects may be 
distinguished from more intensely stained ones by 
virtue of differing saturations. However, converting 

10 from RGB space to HSI space requires more complex 
computation. Conversion to a color ratio is faster ; 
for example, a full image can be converted by the 
ratio technique of the present invention in about 30 
ms while an HSI conversion can take several seconds. 

15 In yet another approach, one could obtain color 

information by taking a single color channel from the 
camera. As an example, consider a blue channel, in 
which objects that are red are relatively dark. 
Objects which are blue, or white, are relatively 

20 light in the blue channel. In principle, one could 
take a single color channel, and simply set a 
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threshold wherein everything darker than some 
threshold is categorized as a candidate object of 
interest, for example, a tumor cell, because it is 
red and hence dark in the channel being reviewed. 
5 However, one problem with the single channel approach 
occurs where illumination is not uniform. Non- 
uniformity of illumination results in non-uniformity 
across the pixel values in any color channel, for 
example, tending to peak in the middle of the image 

10 and dropping off at the edges where the illumination 
falls off. Performing thresholding on this non- 
uniform color information runs into problems, as the 
edges sometimes fall below the threshold, and 
therefore it becomes more difficult to pick the 

15 appropriate threshold level. However, with the ratio 
technique, if the values of the red channel fall off 
from center to edge, then the values of the blue 
channel also fall off center to edge, resulting in a 
uniform ratio, non-uniformities. Thus, the ratio 

20 technique is more immune to illumination. 
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As previously described, the color conversion 
scheme is relatively insensitive to changes in color 
balance, i.e., the relative outputs of the red, 
green, and blue channels. However, some control is 
5 necessary to avoid camera saturation, or inadequate 
exposures in any one of the color bands. This color 
balancing is performed automatically by utilizing a 
calibration slide consisting of a clear area, and a 
"dark" area having a known optical transmission or 

10 density. The system obtains images from the clear 
and "dark/' areas, calculates "white" and "black" 
adjustments for the image processor 25, and thereby 
provides - correct color balance. 

In addition to the color balance control, 

15 certain mechanical alignments are automated in this 
process. The center point in the field of view for 
the various microscope objectives as measured on the 
slide can vary by several (or several tens of) 
microns. This is the result of slight variations in 

20 position of the microscope objectives 44a as 
determined by the turret 44 (Fig. 4), small 
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variations in alignment of the objectives with 
respect to the system optical axis, and other 
factors. Since it is desired that each microscope 
objective be centered at the same point, these 
5 mechanical offsets must be measured and automatically 
compensated. 

This is accomplished by imaging a test slide 
which contains a recognizable feature or mark. An 
image of this pattern is obtained by the system with 

10 a given objective, and the position of the mark 

determined. The system then rotates the turret to 
the next lens objective, obtains an image of the test 
object, and its position is redetermined. -i 1 Apparent 
changes in position of the test mark are recorded for 

15 this objective. This process is continued for all 
objectives. Once these spatial offsets have been 

determined, they are automatically compensated for by 

i 

moving the stage 38 by an equal (but opposite) amount 
of offset during changes in objective. In this way, 
20 as different lens objectives are selected, there is 
no apparent shift in center point or 
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area viewed. A low pass filtering process precedes 
thresholding. An objective of thresholding is to 
obtain a pixel image matrix having only candidate 
objects of interest, such as tumor cells above a 
5 threshold level and everything else below it. 

However, an actual acquired image will contain noise. 
The noise can take several forms, including white 
noise and artifacts. The microscope slide can have 
small fragments of debris that pick up color in the 

10 staining process and these are known as artifacts. 
These artifacts are generally small and scattered 
areas, on the order of a few pixels, which are above 
the threshold. The purpose of low pass filtering is 
to essentially blur or smear the entire color 

15 converted image. The low pass filtering process will 
smear artifacts more than larger objects of interest. 
Such as tumor cells and thereby eliminate or reduce 
the number of artifacts that pass the thresholding 
process. The result is a cleaner thresholded image 

20 downstream. In the low pass filter process, a 3 x 3 
matrix of coefficients is applied to each pixel in 
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the 640 x 480 x-image. A preferred coefficient 



matrix is as follows : 



1/9 1/9 1/9 



1/9 1/9 1/9 



5 



1/9 1/9 1/9 



At each pixel location, a 3 x 3 matrix comprising the 
pixel of interest and its neighbors is multiplied by 
the coefficient matrix and summed to yield a single 
value for the pixel of interest. The output of this 

10 spatial convolution process is again a 640 x 480 
matrix; As an example, consider a case where the 
center ^ pixel and only the center pixel, has a value 
of 255. and each of its other neighbors, top left, 
top, top right and so forth, have values of 0. 

15 This singular white pixel case corresponds to a 

small object. The result of the matrix 
multiplication and addition using the coefficient 
matrix is a value of 1/9 (255) or 28 for the center 
pixel, a value which is below the nominal threshold 

20 of 128. Now consider another case in which all the 
pixels have a value of 255 corresponding to a large 
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object. Performing the low pass filtering operation 
on a 3 x 3 matrix for this case yields a value of 255 
for the center pixel. Thus, large objects retain 
their values while small objects are reduced in 
5 amplitude or eliminated. In the preferred method of 
operation, the low pass filtering process is 
performed on the X image twice in succession. 

In order to separate objects of interest, such 

as a tumor cell in the x image from other objects and 

■ 

10 background, a thresholding operation is performed 

designed to set pixels within cells of interest to a 
value of 255, and all other areas to 0. Thresholding 
ideally yields an image in which cells of interest 
are white and the remainder of the image is black. A 

15 problem one faces in thresholding is where to set the 
threshold level. One cannot simply assume that cells 
of interest are indicated by any pixel value above 
the nominal threshold of 128. A typical imaging 
system may use an incandescent halogen light bulb as 

20 a light source. As the bulb ages, the relative 
amounts of red and blue output can change. The 
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tendency as the bulb ages is for the blue to drop off 
more than the red and the green. To accommodate for 
this light source variation over time, a dynamic 
thresholding process is used whereby the threshold is 
5 adjusted dynamically for each acquired image. Thus, 
for each 640 x 480 image, a single threshold value is 
derived specific to that image. 

As shown in Fig. 18, the basic method is to 
calculate, for each field, the mean X value, and the 

10 standard deviation about this mean at 312. The 
threshold is then set at 314 to the mean plus an 
amount defined by. the product of a (user specified) 
factor and the standard deviation of the color 
converted pixel values. The standard deviation 

15 correlates to the structure and number of objects in 
the image. Preferably, the user specified factor is 
in the range of approximately 1.5 to 2.5. The factor 
is selected to be in the lower end of the range for 
slides in which the stain has primarily remained 

20 within cell boundaries and the factor is selected to 
be in the upper end of the range for slides in which 
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the stain is pervasively present throughout the 
slide. In this way, as areas are encountered on the 
slide with greater or lower background intensities, 
the threshold may be raised or lowered to help reduce 
5 background objects. With this method, the threshold 
changes in step with the aging of the light source 
such that the effects of the aging are canceled out. 
The image matrix resulting at 316 from the 
thresholding step is a binary image of black (0) 5 
10 and white (255) pixels. 

>:<- ; As is often the case with thresholding 
operations such as that described above, some 
: undesired areas will lie above the threshold value 
due to noise, small stained cell fragments, and other 

15 artifacts. It is desired and possible to eliminate 
these artifacts by virtue of their small size 
compared with legitimate cells of interest. 
Morphological processes are utilized to perform this 
function. 

20 Morphological processing is similar to the low 

pass filter convolution process described earlier 
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except that it is applied to a binary image. Similar 
to spatial convolution, the morphological process 
traverses an input image matrix, pixel by pixel, and 
places the processed pixels in an output matrix. 
5 Rather than calculating a weighted sum of neighboring 
pixels as m the low pass convolution process, the 
morphological process uses set theory operations to 
combine neighboring pixels in a nonlinear fashion. 
Erosion is a process whereby a single pixel 

10 layer is taken away from the edge of an object. 

Dilation is the opposite process which adds a single 
pixel layer to the edges of an object. The power of 
morphological processing is that it provides for 
further discrimination to eliminate small objects 

15 that have survived the thresholding process and yet 
are not likely tumor cells. The erosion and dilation 
processes that make up a morphological "open" 
preferably make small objects disappear yet allows 
large objects to remain. Morphological processing of 

20 binary images is described in detail in "Digital 
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Image Processing", pages 127-137, G.A. Baxes, John 
Wiley & Sons (1994) . 

Fig. 19 illustrates the flow diagram for this 
process. As shown here, a morphological "open" 
5 process performs this suppression. A single 

morphological open consists of a . single morphological 
erosion 320 followed by a single morphological 
dilation 322. Multiple "opens" consist of multiple 
erosions followed by multiple dilations. In the 

10 preferred embodiment, one or two morphological opens 
are found to be suitable. At this point in the 
processing chain, the processed image contains 
thresholded objects of interest, such as tumor cells 
(if any were present in the original image) , and 

15 possibly some residual artifacts that were too large 
to be eliminated by the processes above. 

Fig. 20 provides a flow diagram illustrating a 
blob analysis performed to determine the number, 
size, and location of objects in the thresholded 

20 image. A blob is defined as a region of connected 
pixels having the same "color," in this case, a 
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value of 255. Processing is performed over the 
entire image to determine the number of such regions 
at 324 and to determine the area and x,y coordinates 
for each detected blob at 326. Comparison of the 
5 size of each blob to a known minimum area at 328 for 
a tumor cell allows a refinement in decisions about 
which objects are objects of interest, such as tumor 
cells, and which are artifacts. The location 

(x,y coordinates) of objects identified as cells of 

i 

10 interest in this stage are saved for the final 4 Ox 
reimaging step described below. Objects not passing- 
the size test are disregarded as artifacts. :i 

The processing chain described above identifies ■ 
objects at the scanning magnification as cells of 

15 interest candidates. As illustrated in Fig. 21, at 
the completion of scanning, the system switches to 
the 4 Ox magnification objective at 330, and each 
candidate is reimaged to confirm the identification 
332. Each 40x image is reprocessed at 334 using the 

20 same steps as described above but with test 
parameters suitably modified for the higher 
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magnification (e.g., area). At 336, a region of 
interest centered on each confirmed cell is saved to 
the hard drive for review by the pathologist. 

As noted earlier, a mosaic of saved images is 
5 made available for viewing by the pathologist . As 
shown in Fig. 22, a series of images of cells which 
have been confirmed by the image analysis is 
presented in the mosaic 150. The pathologist can 
then visually inspect the images to make a 

10 determination whether to accept (152) or reject (153) 
each cell image. Such a 5r determination can be noted 
and saved with the mosaic of images for generating a 
printed report. r^v * 

In addition to saving the image of the cell and 

15 its region, the cell coordinates are saved should the 
pathologist wish to directly view the cell through 
the oculars or on the image monitor. In this case, 
the pathologist reloads the slide carrier, selects 
the slide and cell for review from a mosaic of cell 

20 images, and the system automatically positions the 
cell under the microscope for viewing. 
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It has been found that normal cells whose nuclei 
have been stained with hematoxylin are often quite 
numerous, numbering in the thousands per lOx image. 
Since these cells are so numerous, and since they 
5 tend to clump, counting each individual nucleated 

cell would add an excessive processing burden, at the 
expense of speed, and would not necessarily provide 
an accurate count due to clumping. The apparatus 
performs an estimation process in which the total 

10 area of each field that is stained hematoxylin blue 
iscmeasured and this area is divided by the average 
size of a nucleated cell. Fig. 23 outlines this 
process. In this process, a single color band (the 
red channel provides the best contrast for blue 

15 stained nucleated cells) is processed by calculating 
the average pixel value for each field at 342, 
establishing two threshold values (high and low) as 
indicated at 344, 346, and counting the number of 
pixels between these two values at 348. In the 

20 absence of dirt, or other opaque debris, this 

provides a count of the number of predominantly blue 
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pixels. By dividing this value by the average area 
for a nucleated cell at 350, and looping over all 
fields at 352 , an approximate cell count is obtained. 
Preliminary testing of this process indicates an 
5 accuracy with +/- 15%. It should be noted that for 
some slide preparation techniques, the size of 
nucleated cells can be significantly larger than the 
typical size. The operator can select the 
appropriate nucleated cell size to compensate for 

10 these characteristics. 

As with any imaging system, there is some loss 
of modulation transfer (i.e., contrast) due to the 
modulation transfer function (MTF) characteristics of 
the imaging optics, camera, electronics, and other 

15 components. Since it is desired to save "high 
quality" images of cells of interest both for 
pathologist review and for archival purposes, it is 
desired to compensate for these MTF losses. An MTF 
compensation, or MTFC, is performed as a digital 

20 process applied to the acquired digital images. A 
digital filter is utilized to restore the high 
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spatial frequency content of the images upon storage, 
while maintaining low noise levels. With this MTPC 
technology, image quality is enhanced, or restored, 
through the use of digital processing methods as 
5 opposed to conventional oil-immersion or other 

hardware based methods. MTFC is described further in 
"The Image Processing Handbook," pages 225 and 337, 
J. C. Rues, CRC Press (1995) . 

Referring to Fig. 24, the functions available in 

10 a user interface of the apparatus 10 are shown. From 
the user interface, which is presented graphically on 
computer monitor 26, an operator can select among 
apparatus functions which include acquisition 402, 
analysts 404, and system configuration 406. At the 

15 acquisition level 402, the operator can select 
between manual 408 and automatic 410 modes of 
operation. In the manual mode, the operator is 
presented with manual operations 409. Patient 
information 414 regarding an assay can be entered at 

20 412. In the analysis level 404, review 416 and 
report 418 functions are made available. At the 
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review level 416, the operator can select a montage 
function 420. At this montage level, a pathologist 
can perform diagnostic review functions including 
visiting an image 422, accept/reject of cells 424, 
5 nucleated cell counting 426, accept/reject of cell 
counts 428, and saving of pages at 430. The report 
level 418 allows an operator to generate patient 
reports 432. In the configuration level 406, the 
operator can select to configure preferences at 434, 

10 input operator information 437 at 436, create a 
system:, log at 438, and toggle a menu panel at 440. 
The configuration preferences include scan area 
selection functions at 442, 452; montage 
specifications at 444, bar code handling at 446, 

15 default cell counting at 448, stain selection at 450, 
and scan objective selection at 454. 
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Computer Implementation 

Aspects of the invention may be implemented in 
hardware or software, or a combination of both. 
However, preferably, the algorithms and processes of 
5 the invention are implemented in one or more computer 
programs executing on programmable computers each 
comprising at least one processor, at least one data 
storage system (including volatile and non- volatile 
memory and/or storage elements) , at least one input 

10 device, and at least one output device. Program code 
is applied to input data to perform the .functions 
described herein and generate output information. 
The output information is applied to one or more 
output devices, in known fashion. 

15 Each program may be implemented in any desired 

computer language (including machine, assembly, high 
level procedural, or object oriented programming 
languages) to communicate with a computer system. In 
any case, the language may be a compiled or 

20 interpreted language. 
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Each such computer program is preferably stored 
on a storage media or device (e.g., ROM, CD-ROM, 
tape, or magnetic diskette) readable by a general or 
special purpose programmable computer, for 
5 configuring and operating the computer when the 
storage media or device is read by the computer to 
perform the procedures described herein. The 
inventive system may also be considered to be 
implemented as a computer-readable storage medium, 

10 configured with a computer program, where the storage 
medium so configured* causes a computer to operate in 
a specific and predefined. manner to perform the 
functions described herein. 

A number of embodiments of the present invention 

15 have been described. Nevertheless, various 

modifications may be made without departing from the 
spirit and scope of the invention. Accordingly, the 
invention is not to be limited by the specific 
illustrated embodiment, but only by the scope of the 

20 appended claims . 
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What is claimed: 



1. A method for automated detection of a cell 
5 proliferative disorder, comprising, 

providing a sample on a slide, wherein the sample is 
stained with an antibody to a protein associated with 
a cell proliferative disorder; 

identifying a processing parameter for the 
10 stained sample; 

.1 scanning the stained sample at a plurality 
oftrlocations at a low magnification on an 
optical system; 

acquiring a low magnification image at the 
15 low magnification at each location in the 

scanned area; 

processing each low magnification image to 
detect a candidate objects of interest; 

storing stage coordinates of each location 
20 for each candidate object of interest; 
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adjusting the optical system to a higher 
magnification; 

repositioning the stage to the location for 
each candidate object of interest; 
5 acquiring a higher magnification image of 

each candidate object of interest; and 

storing each higher magnification image. 

2. The method of claim 1, wherein the cell 
10 proliferative disorder is a neoplasm. 

3. The method of claim 1, wherein the cell 
proliferative disorder is breast cancer. - < 



15 4. The method of claim 1, wherein the antibody is 
selected from the group consisting of an anti- 
HER2/neu antibody, ant i -estrogen receptor 
antibody, ant i -progesterone receptor antibody, 
anti-p53 antibody and anti-cyclin Dl antibody. 



20 
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5 

6. 



10 7. 



15 



The method of claim 1, wherein the processing 
parameter is selected from the group consisting 
of a positive control, negative control or test 
sample . 

The method of claim 1, wherein the detection of 
the candidate object of interest is by size, 
color or shape. 

An automated system for quantification of a cell 
proliferative disorder in a sample, comprising: 

a slide configured to accept a sample, 
wherein the sample is stained with an antibody 
to a protein associated with a cell 
proliferative disorder; 

a code associated with the slide for 
identification of the slide's processing 
parameters ; 

a lens for magnification of the sample on 
the slide; 
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a stage configured to hold the slide, 
wherein the stage is optically coupled to the 
lens ; 

a camera optically coupled to the lens 
5 configured to acquire an image from the lens; 

a computer program residing on a computer- 
readable medium for 

executing the processing parameters of 

the code; 

10 selecting a position on the stage 

identifying a. candidate object of 
interest based upon color, shape or size; 

obtaining a low magnification image of 
the candidate object of interest; 
15 storing the position of the candidate 

object of interest; 

adjusting the magnification of the 

lens ; 

repositioning the stage to the position 
20 of the candidate object of interest; 
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obtaining a higher magnification image 
of the candidate object of interest; 

storing the higher magnification image; 
allowing retrieval of the images 
5 acquired; and 

a monitor configured to view the 

images . 

8. The system of claim 7, wherein the code is a bar 
10 code . 

9. The system of ;claim 7, wherein the lens is an 
objective of a -microscope. 

15 10. The system of claim 7, wherein the camera is a 
CCD camera. 



11. A computer program, residing on a computer- 
readable medium, comprising instructions for 
20 causing a computer to: 

execute a processing parameter of a slide; 
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select a position on a stage of an optical 
system; 

identify a candidate object of interest 
based upon color, shape or size; 
5 obtain a low magnification image of the 

candidate object of interest ; 

store the position of the candidate object 
of interest; 

adjust the magnification of the optical 
10 systems- 
reposition the stage to the position of the 
. ■ candidate object of interest; 
: ^ ' obtain a higher magnification image of the 

candidate object of interest; 
15 store the higher magnification image; and 

allow retrieval of the images acquired. 



12 . An apparatus comprising a computer- readable 
storage medium tangibly embodying program 
20 instruction for quantifying a cell proliferative 

disorder, the program instruction inducing 
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instruction operable for causing an optical 
system to: 

execute a processing parameter of a slide; 

select a position on a stage of an optical 
5 system; 

identify a candidate object of interest 
based upon color, shape or size; 

obtain a low magnification image of the 
candidate object of interest; 
10 store the position of the candidate object 

of interest; 

adjust the magnification of the optical 
system; 

reposition the stage to the position of the 
15 candidate object of interest; 

obtain a higher magnification image of the 
candidate object of interest; 

store the higher magnification image; and 
allow retrieval of the images acquired. 
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13. The apparatus of claim 12, wherein the image is 
a digital image. 
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